Stop recreating the hashmap all the time. #363
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Narsil
force-pushed
the
fix_small_tensors_performance
branch
from
11f6379
to
bc466b5
Compare
Fixes #361 Potentially superseeds #362 Co-Authored-By: Batuhan Taskaya <[email protected]>
|
do you have account on twitter? i would like to follow you good job |
|
https://twitter.com/narsilou |
Followed thanks |
6 tasks
RyanJDick
added a commit
to invoke-ai/InvokeAI
that referenced
this pull request
Oct 11, 2023
## What type of PR is this? (check all applicable) - [ ] Refactor - [ ] Feature - [ ] Bug Fix - [x] Optimization - [ ] Documentation Update - [ ] Community Node Submission ## Description @Millu pointed out this safetensors PR a few weeks ago, which claimed to offer a performance benefit: huggingface/safetensors#362 . It was superseded by huggingface/safetensors#363 and included in the latest [safetensors 0.4.0 release](https://github.com/huggingface/safetensors/releases/tag/v0.4.0). Here are the results from my local performance comparison: ``` Before(0.3.1) / After(0.4.0) sdxl:main:tokenizer from disk to cpu in 0.46s / 0.46s sdxl:main:text_encoder from disk to cpu in 2.12s / 2.32s embroidered_style_v1_sdxl.safetensors:sdxl:lora' from disk to cpu in 0.67s / 0.36s VoxelXL_v1.safetensors:sdxl:lora' from disk to cpu in 1.64s / 0.60s ryan_db_sdxl_epoch640.safetensors:sdxl:lora' from disk to cpu in 2.46s / 1.40s sdxl:main:tokenizer_2 from disk to cpu in 0.37s / 0.39s sdxl:main:text_encoder_2 from disk to cpu in 3.78s / 4.70s sdxl:main:unet from disk to cpu in 4.66s / 3.08s sdxl:main:scheduler from disk to cpu in 0.34s / 0.33s sdxl:main:vae from disk to cpu in 0.66s / 0.51s TOTAL GRAPH EXECUTION TIME: 56.489s / 53.416s ``` The benefit was marginal on my system (maybe even within measurement error), but I figured we might as well pull it.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
This is a different approach from #362 but should fix the same issue in the same way.
This doesn't change any external python API, and makes every existing code faster (by just using the structurs smarter).
This is better IMO as it fixes every potential use for the
safe_openmethod, whereas #362 introduces a new layer that has to be used in order to benefit (so only users ofsafetensors.torch.load_filewould actually see the difference).The crux of the problem was
Metadata::tensorswhich is a method that creates a fullHashMap<String, String>to access the tensors. The structure used to be exactly that and just give directly the structure to callers, therefore relatively free. But since, the structure was refactored into aHashMap<String, usize>andVec<TensorInfo>.The main reason for it, is that the
Vecis intrinsically ordered, and we maintain thedata_offsetsordered in order to get efficient comparison when checking for malformed files (with overlapping data_offsets).It also helps in during the serialization where we already have those information sorted so we write them sorted into the JSON header.
Fixes #361
Potentially superseeds #362
Fixes # (issue) or description of the problem this PR solves.